{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control    (Actor-Critic Reinforcement Learning)\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Reacher_Windows_x86_64/Reacher.exe')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from  collections  import deque\n",
    "from itertools import count\n",
    "import torch\n",
    "import time\n",
    "from ddpg_agent import Agent\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1, Score: 0.71, Max: 2.04, Min: 0.02 \n",
      "Episode: 2, Score: 0.80, Max: 2.12, Min: 0.10 \n",
      "Episode: 3, Score: 0.76, Max: 1.59, Min: 0.00 \n",
      "Episode: 4, Score: 0.72, Max: 1.46, Min: 0.13 \n",
      "Episode: 5, Score: 1.17, Max: 2.48, Min: 0.26 \n",
      "Episode: 6, Score: 0.70, Max: 1.75, Min: 0.14 \n",
      "Episode: 7, Score: 1.12, Max: 2.21, Min: 0.09 \n",
      "Episode: 8, Score: 1.13, Max: 2.09, Min: 0.15 \n",
      "Episode: 9, Score: 1.26, Max: 2.90, Min: 0.00 \n",
      "Episode: 10, Score: 1.09, Max: 2.44, Min: 0.13 \n",
      "*** Episode 10\tAverage Score: 0.95, Time: 00:02:42 *** \n",
      "Episode: 11, Score: 1.88, Max: 3.97, Min: 0.31 \n",
      "Episode: 12, Score: 1.51, Max: 2.79, Min: 0.18 \n",
      "Episode: 13, Score: 1.51, Max: 3.66, Min: 0.21 \n",
      "Episode: 14, Score: 1.43, Max: 2.62, Min: 0.39 \n",
      "Episode: 15, Score: 1.78, Max: 3.42, Min: 0.75 \n",
      "Episode: 16, Score: 1.93, Max: 3.17, Min: 0.90 \n",
      "Episode: 17, Score: 1.90, Max: 3.69, Min: 0.73 \n",
      "Episode: 18, Score: 1.97, Max: 4.75, Min: 0.45 \n",
      "Episode: 19, Score: 1.95, Max: 4.05, Min: 0.16 \n",
      "Episode: 20, Score: 1.91, Max: 4.26, Min: 0.16 \n",
      "*** Episode 20\tAverage Score: 1.36, Time: 00:05:31 *** \n",
      "Episode: 21, Score: 1.96, Max: 3.98, Min: 0.52 \n",
      "Episode: 22, Score: 2.28, Max: 3.93, Min: 0.38 \n",
      "Episode: 23, Score: 2.47, Max: 5.23, Min: 0.64 \n",
      "Episode: 24, Score: 2.27, Max: 6.58, Min: 0.53 \n",
      "Episode: 25, Score: 2.36, Max: 3.80, Min: 0.54 \n",
      "Episode: 26, Score: 2.76, Max: 4.68, Min: 1.11 \n",
      "Episode: 27, Score: 3.21, Max: 5.76, Min: 1.26 \n",
      "Episode: 28, Score: 3.72, Max: 5.98, Min: 1.74 \n",
      "Episode: 29, Score: 3.72, Max: 8.61, Min: 1.60 \n",
      "Episode: 30, Score: 3.54, Max: 5.42, Min: 1.83 \n",
      "*** Episode 30\tAverage Score: 1.85, Time: 00:08:33 *** \n",
      "Episode: 31, Score: 4.06, Max: 6.46, Min: 1.91 \n",
      "Episode: 32, Score: 4.48, Max: 6.67, Min: 2.78 \n",
      "Episode: 33, Score: 4.61, Max: 8.28, Min: 1.54 \n",
      "Episode: 34, Score: 4.35, Max: 9.52, Min: 1.05 \n",
      "Episode: 35, Score: 4.73, Max: 6.91, Min: 3.07 \n",
      "Episode: 36, Score: 5.08, Max: 9.59, Min: 1.78 \n",
      "Episode: 37, Score: 4.67, Max: 8.35, Min: 2.13 \n",
      "Episode: 38, Score: 3.87, Max: 6.79, Min: 0.83 \n",
      "Episode: 39, Score: 5.08, Max: 14.40, Min: 1.86 \n",
      "Episode: 40, Score: 4.47, Max: 6.14, Min: 1.73 \n",
      "*** Episode 40\tAverage Score: 2.52, Time: 00:12:13 *** \n",
      "Episode: 41, Score: 4.76, Max: 8.42, Min: 2.32 \n",
      "Episode: 42, Score: 5.85, Max: 10.02, Min: 2.01 \n",
      "Episode: 43, Score: 4.86, Max: 8.24, Min: 1.36 \n",
      "Episode: 44, Score: 5.56, Max: 8.69, Min: 4.16 \n",
      "Episode: 45, Score: 5.86, Max: 11.20, Min: 2.66 \n",
      "Episode: 46, Score: 5.50, Max: 8.03, Min: 2.99 \n",
      "Episode: 47, Score: 5.51, Max: 8.64, Min: 3.27 \n",
      "Episode: 48, Score: 6.51, Max: 9.83, Min: 3.18 \n",
      "Episode: 49, Score: 5.97, Max: 7.99, Min: 3.57 \n",
      "Episode: 50, Score: 6.85, Max: 13.71, Min: 3.88 \n",
      "*** Episode 50\tAverage Score: 3.16, Time: 00:15:53 *** \n",
      "Episode: 51, Score: 6.53, Max: 9.68, Min: 3.14 \n",
      "Episode: 52, Score: 7.99, Max: 16.33, Min: 3.70 \n",
      "Episode: 53, Score: 7.04, Max: 10.94, Min: 3.69 \n",
      "Episode: 54, Score: 7.78, Max: 10.47, Min: 5.41 \n",
      "Episode: 55, Score: 8.55, Max: 12.90, Min: 5.83 \n",
      "Episode: 56, Score: 8.47, Max: 15.28, Min: 4.61 \n",
      "Episode: 57, Score: 8.42, Max: 11.66, Min: 5.74 \n",
      "Episode: 58, Score: 8.30, Max: 12.44, Min: 3.46 \n",
      "Episode: 59, Score: 7.26, Max: 10.25, Min: 1.86 \n",
      "Episode: 60, Score: 7.76, Max: 12.65, Min: 3.42 \n",
      "*** Episode 60\tAverage Score: 3.94, Time: 00:19:40 *** \n",
      "Episode: 61, Score: 9.27, Max: 12.87, Min: 6.00 \n",
      "Episode: 62, Score: 7.94, Max: 14.42, Min: 5.63 \n",
      "Episode: 63, Score: 7.71, Max: 15.13, Min: 4.35 \n",
      "Episode: 64, Score: 8.10, Max: 13.90, Min: 2.92 \n",
      "Episode: 65, Score: 8.86, Max: 12.54, Min: 5.28 \n",
      "Episode: 66, Score: 9.26, Max: 13.45, Min: 4.51 \n",
      "Episode: 67, Score: 8.43, Max: 15.48, Min: 4.05 \n",
      "Episode: 68, Score: 9.31, Max: 14.45, Min: 4.41 \n",
      "Episode: 69, Score: 9.57, Max: 14.37, Min: 5.75 \n",
      "Episode: 70, Score: 10.49, Max: 17.76, Min: 6.09 \n",
      "*** Episode 70\tAverage Score: 4.65, Time: 00:23:29 *** \n",
      "Episode: 71, Score: 9.73, Max: 16.96, Min: 2.50 \n",
      "Episode: 72, Score: 9.23, Max: 15.43, Min: 5.28 \n",
      "Episode: 73, Score: 9.34, Max: 12.93, Min: 4.95 \n",
      "Episode: 74, Score: 9.48, Max: 14.23, Min: 2.60 \n",
      "Episode: 75, Score: 10.19, Max: 18.09, Min: 5.57 \n",
      "Episode: 76, Score: 8.83, Max: 16.50, Min: 2.15 \n",
      "Episode: 77, Score: 11.30, Max: 31.48, Min: 5.08 \n",
      "Episode: 78, Score: 11.26, Max: 19.91, Min: 4.80 \n",
      "Episode: 79, Score: 10.47, Max: 15.64, Min: 1.81 \n",
      "Episode: 80, Score: 12.48, Max: 25.26, Min: 6.37 \n",
      "*** Episode 80\tAverage Score: 5.34, Time: 00:27:15 *** \n",
      "Episode: 81, Score: 9.90, Max: 15.15, Min: 0.00 \n",
      "Episode: 82, Score: 13.04, Max: 37.81, Min: 5.46 \n",
      "Episode: 83, Score: 12.24, Max: 15.81, Min: 7.61 \n",
      "Episode: 84, Score: 11.99, Max: 16.75, Min: 3.53 \n",
      "Episode: 85, Score: 11.20, Max: 16.94, Min: 2.18 \n",
      "Episode: 86, Score: 12.56, Max: 17.36, Min: 7.74 \n",
      "Episode: 87, Score: 12.38, Max: 30.75, Min: 2.89 \n",
      "Episode: 88, Score: 12.60, Max: 22.23, Min: 6.41 \n",
      "Episode: 89, Score: 12.16, Max: 25.83, Min: 2.36 \n",
      "Episode: 90, Score: 12.74, Max: 19.56, Min: 6.18 \n",
      "*** Episode 90\tAverage Score: 6.09, Time: 00:31:03 *** \n",
      "Episode: 91, Score: 15.16, Max: 22.37, Min: 8.05 \n",
      "Episode: 92, Score: 16.26, Max: 31.84, Min: 8.09 \n",
      "Episode: 93, Score: 15.31, Max: 20.67, Min: 1.89 \n",
      "Episode: 94, Score: 15.67, Max: 22.35, Min: 6.24 \n",
      "Episode: 95, Score: 16.77, Max: 22.11, Min: 8.84 \n",
      "Episode: 96, Score: 14.93, Max: 26.66, Min: 1.62 \n",
      "Episode: 97, Score: 16.58, Max: 22.84, Min: 8.56 \n",
      "Episode: 98, Score: 16.64, Max: 21.95, Min: 10.09 \n",
      "Episode: 99, Score: 19.01, Max: 39.24, Min: 7.93 \n",
      "Episode: 100, Score: 17.08, Max: 24.32, Min: 3.53 \n",
      "*** Episode 100\tAverage Score: 7.12, Time: 00:34:51 *** \n",
      "Episode: 101, Score: 17.95, Max: 22.91, Min: 11.00 \n",
      "Episode: 102, Score: 16.56, Max: 24.24, Min: 1.67 \n",
      "Episode: 103, Score: 21.07, Max: 38.31, Min: 13.45 \n",
      "Episode: 104, Score: 18.64, Max: 27.59, Min: 4.35 \n",
      "Episode: 105, Score: 20.56, Max: 25.43, Min: 14.37 \n",
      "Episode: 106, Score: 19.86, Max: 26.74, Min: 13.93 \n",
      "Episode: 107, Score: 19.58, Max: 29.77, Min: 12.95 \n",
      "Episode: 108, Score: 20.27, Max: 35.10, Min: 10.09 \n",
      "Episode: 109, Score: 20.81, Max: 29.72, Min: 4.26 \n",
      "Episode: 110, Score: 21.52, Max: 28.49, Min: 11.96 \n",
      "*** Episode 110\tAverage Score: 8.99, Time: 00:38:39 *** \n",
      "Episode: 111, Score: 20.78, Max: 32.40, Min: 10.49 \n",
      "Episode: 112, Score: 21.36, Max: 29.38, Min: 14.16 \n",
      "Episode: 113, Score: 21.41, Max: 39.38, Min: 10.43 \n",
      "Episode: 114, Score: 23.81, Max: 30.33, Min: 16.39 \n",
      "Episode: 115, Score: 25.42, Max: 38.99, Min: 18.17 \n",
      "Episode: 116, Score: 23.45, Max: 32.54, Min: 14.20 \n",
      "Episode: 117, Score: 24.80, Max: 39.39, Min: 13.74 \n",
      "Episode: 118, Score: 25.33, Max: 39.52, Min: 9.36 \n",
      "Episode: 119, Score: 24.78, Max: 39.53, Min: 12.81 \n",
      "Episode: 120, Score: 26.61, Max: 33.77, Min: 16.30 \n",
      "*** Episode 120\tAverage Score: 11.19, Time: 00:42:28 *** \n",
      "Episode: 121, Score: 24.61, Max: 31.65, Min: 12.96 \n",
      "Episode: 122, Score: 25.64, Max: 33.35, Min: 14.13 \n",
      "Episode: 123, Score: 25.77, Max: 32.62, Min: 14.79 \n",
      "Episode: 124, Score: 26.46, Max: 39.47, Min: 11.65 \n",
      "Episode: 125, Score: 26.07, Max: 31.84, Min: 17.42 \n",
      "Episode: 126, Score: 25.32, Max: 38.31, Min: 16.42 \n",
      "Episode: 127, Score: 26.99, Max: 33.45, Min: 20.29 \n",
      "Episode: 128, Score: 26.67, Max: 34.61, Min: 18.10 \n",
      "Episode: 129, Score: 26.91, Max: 37.99, Min: 14.14 \n",
      "Episode: 130, Score: 25.93, Max: 38.60, Min: 6.53 \n",
      "*** Episode 130\tAverage Score: 13.51, Time: 00:46:15 *** \n",
      "Episode: 131, Score: 27.38, Max: 32.30, Min: 21.22 \n",
      "Episode: 132, Score: 27.39, Max: 35.68, Min: 15.50 \n",
      "Episode: 133, Score: 29.01, Max: 37.08, Min: 18.37 \n",
      "Episode: 134, Score: 27.32, Max: 35.65, Min: 18.57 \n",
      "Episode: 135, Score: 28.04, Max: 34.76, Min: 19.73 \n",
      "Episode: 136, Score: 29.73, Max: 33.93, Min: 23.85 \n",
      "Episode: 137, Score: 29.27, Max: 33.62, Min: 18.22 \n",
      "Episode: 138, Score: 32.11, Max: 38.77, Min: 18.49 \n",
      "Episode: 139, Score: 30.12, Max: 34.20, Min: 19.54 \n",
      "Episode: 140, Score: 30.16, Max: 33.26, Min: 24.46 \n",
      "*** Episode 140\tAverage Score: 15.96, Time: 00:50:02 *** \n",
      "Episode: 141, Score: 30.81, Max: 39.37, Min: 16.92 \n",
      "Episode: 142, Score: 29.94, Max: 37.91, Min: 20.18 \n",
      "Episode: 143, Score: 31.89, Max: 38.72, Min: 24.29 \n",
      "Episode: 144, Score: 32.81, Max: 36.73, Min: 27.04 \n",
      "Episode: 145, Score: 31.48, Max: 36.08, Min: 26.59 \n",
      "Episode: 146, Score: 32.75, Max: 36.67, Min: 27.19 \n",
      "Episode: 147, Score: 31.96, Max: 37.55, Min: 26.55 \n",
      "Episode: 148, Score: 32.50, Max: 38.13, Min: 25.44 \n",
      "Episode: 149, Score: 32.21, Max: 37.37, Min: 23.54 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 150, Score: 31.09, Max: 36.01, Min: 20.85 \n",
      "*** Episode 150\tAverage Score: 18.56, Time: 00:53:50 *** \n",
      "Episode: 151, Score: 30.04, Max: 35.32, Min: 22.66 \n",
      "Episode: 152, Score: 32.84, Max: 36.18, Min: 25.95 \n",
      "Episode: 153, Score: 32.63, Max: 36.52, Min: 24.48 \n",
      "Episode: 154, Score: 35.48, Max: 39.50, Min: 25.82 \n",
      "Episode: 155, Score: 34.71, Max: 38.77, Min: 29.35 \n",
      "Episode: 156, Score: 34.20, Max: 39.17, Min: 27.48 \n",
      "Episode: 157, Score: 36.20, Max: 39.27, Min: 22.97 \n",
      "Episode: 158, Score: 36.65, Max: 39.30, Min: 33.21 \n",
      "Episode: 159, Score: 35.61, Max: 38.84, Min: 27.36 \n",
      "Episode: 160, Score: 35.07, Max: 39.04, Min: 28.34 \n",
      "*** Episode 160\tAverage Score: 21.22, Time: 00:57:38 *** \n",
      "Episode: 161, Score: 36.24, Max: 39.57, Min: 29.96 \n",
      "Episode: 162, Score: 35.79, Max: 39.55, Min: 29.40 \n",
      "Episode: 163, Score: 36.97, Max: 39.54, Min: 29.14 \n",
      "Episode: 164, Score: 35.53, Max: 39.65, Min: 21.76 \n",
      "Episode: 165, Score: 35.76, Max: 39.02, Min: 27.52 \n",
      "Episode: 166, Score: 37.04, Max: 39.62, Min: 31.07 \n",
      "Episode: 167, Score: 36.38, Max: 39.30, Min: 25.66 \n",
      "Episode: 168, Score: 36.98, Max: 39.48, Min: 25.20 \n",
      "Episode: 169, Score: 36.67, Max: 39.61, Min: 28.32 \n",
      "Episode: 170, Score: 38.70, Max: 39.66, Min: 35.71 \n",
      "*** Episode 170\tAverage Score: 23.99, Time: 01:01:27 *** \n",
      "Episode: 171, Score: 35.93, Max: 39.45, Min: 25.32 \n",
      "Episode: 172, Score: 37.77, Max: 39.54, Min: 32.79 \n",
      "Episode: 173, Score: 37.36, Max: 39.40, Min: 33.56 \n",
      "Episode: 174, Score: 36.26, Max: 39.14, Min: 32.26 \n",
      "Episode: 175, Score: 37.81, Max: 39.20, Min: 36.06 \n",
      "Episode: 176, Score: 37.71, Max: 39.45, Min: 32.99 \n",
      "Episode: 177, Score: 38.18, Max: 39.52, Min: 33.94 \n",
      "Episode: 178, Score: 36.87, Max: 39.55, Min: 34.29 \n",
      "Episode: 179, Score: 37.56, Max: 39.53, Min: 32.90 \n",
      "Episode: 180, Score: 37.20, Max: 39.42, Min: 33.03 \n",
      "*** Episode 180\tAverage Score: 26.69, Time: 01:05:14 *** \n",
      "Episode: 181, Score: 37.42, Max: 39.45, Min: 32.59 \n",
      "Episode: 182, Score: 36.25, Max: 39.34, Min: 30.82 \n",
      "Episode: 183, Score: 37.19, Max: 39.49, Min: 31.03 \n",
      "Episode: 184, Score: 37.23, Max: 39.47, Min: 32.66 \n",
      "Episode: 185, Score: 37.10, Max: 39.50, Min: 34.44 \n",
      "Episode: 186, Score: 36.28, Max: 39.19, Min: 30.47 \n",
      "Episode: 187, Score: 36.13, Max: 39.39, Min: 33.07 \n",
      "Episode: 188, Score: 34.60, Max: 37.87, Min: 29.54 \n",
      "Episode: 189, Score: 35.65, Max: 39.04, Min: 24.96 \n",
      "Episode: 190, Score: 35.49, Max: 39.23, Min: 29.52 \n",
      "*** Episode 190\tAverage Score: 29.12, Time: 01:09:03 *** \n",
      "Episode: 191, Score: 36.78, Max: 39.39, Min: 33.28 \n",
      "Episode: 192, Score: 36.76, Max: 39.52, Min: 32.55 \n",
      "Episode: 193, Score: 37.60, Max: 39.50, Min: 34.84 \n",
      "Episode: 194, Score: 38.00, Max: 39.30, Min: 35.72 \n",
      "Episode: 195, Score: 38.69, Max: 39.49, Min: 36.85 \n",
      "*** Episode 195\tAverage Score: 30.20, Time: 01:10:57 *** \n",
      "Environment solved !\n"
     ]
    }
   ],
   "source": [
    "def ddpg(n_episodes=1000, print_every=10):\n",
    "    scores_deque = deque(maxlen=100)\n",
    "    scores_global = []\n",
    "        \n",
    "    time_start = time.time()\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]      # reset the environment    \n",
    "        states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "        scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "        agent.reset()\n",
    "        score_average = 0\n",
    "\n",
    "        for timestep in range(2000):\n",
    "            actions = agent.act(states)\n",
    "            env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "            next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "            rewards = env_info.rewards                         # get reward (for each agent)\n",
    "            dones = env_info.local_done                        # see if episode finished\n",
    "            agent.step(states, actions, rewards, next_states, dones, timestep)\n",
    "            states = next_states                               # roll over states to next time step\n",
    "            scores += rewards                                  # update the score (for each agent)    \n",
    "            if np.any(dones):                                  # exit loop if episode finished\n",
    "                break\n",
    "        \n",
    "        score = np.mean(scores)\n",
    "        scores_deque.append(score)\n",
    "        score_average = np.mean(scores_deque)\n",
    "        scores_global.append(score)\n",
    "        \n",
    "        print('Episode: {}, Score: {:.2f}, Max: {:.2f}, Min: {:.2f} '\\\n",
    "              .format(i_episode, score, np.max(scores), np.min(scores)))\n",
    "        \n",
    "        if i_episode % print_every == 0 or (len(scores_deque) == 100 and np.mean(scores_deque) >= 30) :\n",
    "            torch.save(agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            s = (int)(time.time() - time_start) \n",
    "            print('*** Episode {}\\tAverage Score: {:.2f}, Time: {:02}:{:02}:{:02} *** '\\\n",
    "                  .format(i_episode, np.mean(scores_deque), s//3600, s%3600//60, s%60))  \n",
    "            \n",
    "        if len(scores_deque) == 100 and np.mean(scores_deque) >= 30:  \n",
    "            print('Environment solved !')\n",
    "            break\n",
    "            \n",
    "    return scores_global\n",
    "\n",
    "scores = ddpg()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xd8W+XZ+P/PJXlvO96xE9tZTsiOE0JD2COMsgpllULhaUofKKX02zK66K/t83RC24cywiiUUqBllL1JSFIgiROyndixs7zteA/ZlnT//tCxsROvhEiyrev9evll6dY5OpePZF26x7lvMcaglFIqcNn8HYBSSin/0kSglFIBThOBUkoFOE0ESikV4DQRKKVUgNNEoJRSAU4TgVJKBThNBEopFeA0ESilVIAL8vYBRMQO5ANlxpgLRSQbeA5IADYB1xljOgd7jsTERJOVleXtUJVSakzZuHFjrTEmaajtvJ4IgO8CBUCMdf83wP3GmOdE5GHgJuChwZ4gKyuL/Px870aplFJjjIjsH852Xm0aEpEM4ALgMeu+AGcAL1ibPAVc4s0YlFJKDc7bfQR/BH4IuK3744AGY4zTul8KjPdyDEoppQbhtUQgIhcC1caYjb2L+9m03+lPRWS5iOSLSH5NTY1XYlRKKeXdGsES4CIR2Yenc/gMPDWEOBHp7pvIAMr729kYs8IYk2eMyUtKGrKvQyml1DHyWiIwxtxtjMkwxmQBVwEfGmOuBVYCl1ubXQ+84q0YlFJKDc0f1xHcCdwhInvw9Bk87ocYlFJKWXwxfBRjzCpglXW7BFjki+MqpZQaml5ZrJQKWHWtnby2pd9uSr9zdLm4793dVDY6vH4sTQRKqYD1zKf7+c6zn1Hb0uHvUI7w6pZy/vzhHkpqWrx+LE0ESqmAVVLbCkBFg/e/dR8NYwxPrN1Lbmo0J00a5/XjaSJQSgWsnkTQ2D7kth8X1/K95zfjcvd76dNx9UnJIXZVNnPjkmw8EzJ4lyYCpVRAMsaw12p2qWwavEbQ7Ojijue38PJnZeyubPZ6XA+tKiYhMoSL5qZ79VjdfDJqSCmlRpq61k6aHJ7ZbioaHdS3dvLw6mJuOX0yMWHBALyyuYx/5h/EJtKTLDbur2NGesyAz/tFPfXxPtYU1fKzL88gLNjuteP0pjUCpVRA2ms1CwFUNjp4r6CKRz4q4e4Xt2GMp/nnhY2lfFpSx5qiWr6xJIuk6FDy99fT7OjiqY/34ehyHdeYCqua+Z83d3FmbjI3fCnruD73YLRGoJQKSN39A0nRoVQ0trOn2tNM9Ma2CpZuSOTKhZlsL2vkigUZ3H7WVJKiQ6lsdLBxfz0PrSrmwVXFrN9Xx7WLJvDQR8X84uKZZCVGfqGYnv5kPyLwuyvm+KRvoJsmAqVUQNpb20qwXcibGM+uymaKqprJTY0mMjSIB1cVs3RqEvVtXZwwPpbU2DAAFkyM563tlfztk/0kRYfyxtYK3thaAcArm8u57czJ3PdeIZsPNtDlcnPZ/AwunptOaNDQTTydTjevby3nnBNSSYgM8erffjhtGlJKjSl1rZ28tKm0T9nja/dy37u7ASiqaualTaWU1LSQmRBBRnw4FY3tFFa1MDUlmovmpHOgro03rQ/4mb36A/KyEgBo6XDywNXz+PZpk7hu8USmpUTzcXEt28ua+L8P91DV5KC6uYMfvrCVX75eMKy4VxfWUN/WxaXzfNNB3JvWCJRSY8oTa/fywMo9zM2MIycpqqesorGdy+ZncNtzmymoaCLYLpw6NYnU2HAcXW7KGtq5amEmS6ckAvDY2hLsNmF62ueJ4IT0GMKCbUxKimJRdgIn5njG+P/PmwU8+Z99vPxZGXab8Pzyk4iLCOb25zfz8mdl/OiC6QN2/FY2Onh9aznv7KgkITKEpVN8P9uy1giUUqNKUVUzV6/4lKoBhnx+UnIIgK2ljQCUN7RT1tCO28C3nt5IQUUTM9Ji6HIZshMjSY0J69l3SkoU2YmRZMSHU9XUweSkqD4f4MF2G3+8ch6/vXx2nzb8kyaNo9Pl5u+f7ueknHHER4YgIlyZl0lLh5N3d1YN+Pc8/FExv3yjgA376rls3niC7b7/WNZEoJQaNZwuN3f8cwuflBzqaZvvrbXDyZaDDQBstn5v2FcHwJyMWHZXNZOdGMm/b1nCTy+cwbUnTuxp/weYnByNiHDKVM+38hPGHzlMdNnMVE5Ij+1TtjArgSCb0Olyc+7M1J7yxTnjSI8NO6KpqrcN++o4MTuB9fecyd3nTx/uqTiuNBEopUaNFWtK2FbWSESInZW7q3vKPy05xO/f2c2GfXU43YawYBtbSz9PBFGhQfzuijmEBtm44+yphATZuPHkbLISI0mzEkGwXcgaFwHAKVbz0MzDPvAHEhUaxJzMOETg3BkpPeU2m3DJvPGsLqxhxepiWjucffZrdnRRUNHE4pxxJMeEYbf5bqRQb9pHoJQaNZ759ABLpySSmxrNUx/vp63TSURIEI+uLuGDXdVMSookyCZcNj+DFzeW0uVys2FvPfMmxDE1JZotPzvniLb6pOhQbAI5iVEEWc0yp01L5usnTeSC2WnDju1bp+Swo7yJ5F5NTQA3npzNltIG/ufNXWwpbeQv18zveWzTgQbcxlOj8CetESilRoX2ThdlDe0szErg9GnJdLrcfLznEE6Xm3V7Pc0/xTWtzM2MY3HOODqcbjbsrWN3VTOLrA/a/jpsg+020mLDmZYa3VMWFmzn/7t4JimHfagP5pwTUvne2VOPKE+MCuWZ/1rMZfPH80nxoZ6L1QDy99VhtwlzJ8QN+zjeoDUCpdSo0H0lcHZiJHlZCUSG2PlwdzXjokJo6XDyw2XT+MuHezg9N5m5GZ4P1jtf2grA4iFm8Fzx9QVeH7s/NzOOlzaVUdHoYP+hNjbsq2N1US0z0mKICvXvR7HXji4iYcBqINQ6zgvGmJ+JyJPAqUCjtekNxpjN3opDKTU6fVJ8iA93VXHP+dMRkZ5EkJMUSUiQjdNyk3l9SzmRIZ5v+V/Ny+SaRROICg3CbhMSIkMoq2/nh8umkTcxftBjHd756w0zx3uOsb2skYc+KuazA54+jG8syfL6sYfizTTUAZxhjGkRkWBgrYi8ZT32A2PMC148tlJqlHtlcxnPbTjIhbPTmZMZ17NAS7Y1jcNtZ0zhrW0VPGbN258YFdpn/79cM5+wYBvzJgyeBHxlemoMNoG1e2rZfLCBy+aPJzzYzrUnTvB3aN5LBMbTENa9tE6w9eP9ibyVUmNC92yfz+cfZE5mHHtrW0mLDSMixPOxNS01mq/mZfLchoMszjmy6ccXC7ocjfAQO5OTo3h+w0GMgesWTxwxScqrncUiYheRzUA18J4xZp310K9EZKuI3C8ioYM8hVIqQHWv1fvq5nLaOp0U17aSk9R3Urc7zp7KrPGxPpu3/4uamR5Lh9NNfEQwszP820Hcm1cTgTHGZYyZC2QAi0RkJnA3kAssBBKAO/vbV0SWi0i+iOTX1NR4M0yl1AhU1eRgWko0LR1OXt9SQUlNCzmJUX22SY4J47XvnMz8EfLNeijd/QSnTE3y2zUD/fHJ8FFjTAOwClhmjKkwHh3AX4FFA+yzwhiTZ4zJS0ry/dwbSin/cXS5qG/r4oLZacxIi+E3b++i2eHs6R8YreZkemoBZ+Qm+zmSvryWCEQkSUTirNvhwFnALhFJs8oEuATY7q0YlFKjU3VTBwCpsWHce9EJHGrtBDiiaWi0mT8hjueXL+bLs0dWU5Y3awRpwEoR2QpswNNH8DrwjIhsA7YBicAvvRiDUmoU6u4oTo0JY1F2AhdbfQCTkqIG223EExFOzBmHbQQ1C4F3Rw1tBeb1U36Gt46plBobehKBNQ/QLy6Zyfmz0shMiPBnWGOWTjGhlBpxqqwRQ91TPMSEBXPuCamD7aK+AE0ESqkRp6LRQUSInZgwnQXHFzQRKKVGnKomB6kxYT5dwD2QaSJQSo04lU2Oo5r5U30xmgiUUiNOZaOjz8phyrs0ESilvKKy0cE1j35KWUM7xhie/mQfNc0dg+5z33uFXPTAWqqbtUbgS5oIlFJe8dJnpXxcfIj3d1axs6KJn7yyg7+s3ANARWM7ji7XEfv8Z08tW0sb6XKZniUklfdpl7xSyive2lYJwMb99T1lL39WxjeWZHHen9aQkxTJ0zeeSHyvBWEqGx2cNT2ZxTnjuGTueJ/HHKi0RqCUOmqOLheN7V0DPn6wro1tZY0E2YSN++tZv6+OIJvQ2N7FdY+vx+kyFFa1cM1j6+hwemoGbrehqsnBlJRo/mtpDrERwb76cwKeJgKl1FH75Rs7uebRTwd8/O3tntrA1xZPpKyhndW7a1g2M5X02DAO1LXxjZOz+N3lsymoaGJdiWe94UOtnTjdhlTtG/A5TQRKqaNWWNVCQUUTnU53v4+/t7OKE9JjuGSep3mnucPJiTnjuGFJFumxYdxy+mTOmZFKSJCN1YWeaea71x/Q0UK+p30ESqmjVtXkwG3gQF0rk5Oj+zxmjKGgsolL5o5nRloMoUE2OpxuFmUlMC01mm8uzem5UGxRVgKri6xEYM0vpJ3Evqc1AqXUUTHG9Hx7L65pPeLxmpYOmh1OJlmLzM/JiCM2PJgpyZ6ZQ3tfLXzK1EQKq1oob2insrEdQJuG/EATgVLqqDS0ddFhNQmV9JMIussmWR/8d543jd98ZXa/Uy+fMtWz6NSaohoqGh0E2YRxUbp6ra9p05BS6qh0N+EA7K1tOeLx4hpPWY61dsCCiQkDPte0lGhSYkJZXVhLaLCNlJiwEbWEY6DQGoFS6qh0J4LwYHufGsHHxbVUNLZTUtNKeLCdtGE08YgIp0xJYu2eWsrq20mJ0dqAP2giUEodle61AvKy4imp9SSC0vo2rnt8Pb94fSfFNS3kJEUOexWuU6Ym0djexcb99aTFhnstbjUwb65ZHCYi60Vki4jsEJGfW+XZIrJORIpE5HkRCRnquZRSI0dlkwMRWJwzjrrWThraOnlszV5cbsP7BdXsKG/qaRYajpMnJyKC5xoCHTHkF96sEXQAZxhj5gBzgWUishj4DXC/MWYKUA/c5MUYlFLHWWWjg3GRoUxL8Qwbzd9Xz3MbDnBCegydTjc1zR1MOopF5uMjQ5idEQfoiCF/8VoiMB7dPUnB1o8BzgBesMqfAi7xVgxKqeOvsslBamwoOdaH/TefzsfR5eZPV80lI97TtHO0i8yfOiUR0IvJ/MWrfQQiYheRzUA18B5QDDQYY5zWJqWAziyl1ChS2ehZPSxrXCQ3nzqJ5Utz+NuNi5icHM2X56QD9CSJ4TpvVhpRoUHMSI/xRshqCF4dPmqMcQFzRSQOeBmY3t9m/e0rIsuB5QATJkzwWoxKqeH50/tFiHiuKl4wMR6bTbjrvNw+23xzaQ6JUaHMSDu6D/TpaTFsu/ccXZrST3xyHYExpkFEVgGLgTgRCbJqBRlA+QD7rABWAOTl5fWbLJRSvvPMuv1UWwvLDNSWnxAZwk0nZx/T82sS8B9vjhpKsmoCiEg4cBZQAKwELrc2ux54xVsxKKWOj0MtHT1JACBF2/LHFG/2EaQBK0VkK7ABeM8Y8zpwJ3CHiOwBxgGPezEGpdRxsLuyGYBbT59MeLD9qJt+1MjmtaYhY8xWYF4/5SXAIm8dVyn1xT2/4QB/er+Ij354OsF2GwVWIrhhSRa3nzWFILteizqW6KuplDrCmqJayhsd7LWuHN5d2URiVAiJUaGaBMYgfUWVUkfYWdEEQIH1e1dlM7mp2hw0VmkiUEr10dbp7FUTaMblNuyubCY3NXqIPdVopYlAKdXH7spmjDVge1dlM/sPtdLhdDNNE8GYpYlAKdVHd7PQgonx7K5sZktpA+C56EuNTZoIlFJ97CxvIjosiDOnJ1PW0M7ja/eSmRCuiWAM00SglOqjoKKJ6WkxPR/828uauPbEibpy2BimiUApBcCGfXX88IUt7ChvYkZaTE/ncEiQja/mZfo5OuVNumaxUgqA//twD5+WHCIjPpxlM1NJjQkjNSaMU6YmkhCp60eNZZoIlAow//6sjMyECBZMjMfpcuMyhhC7ja2lDVw6dzy/uXx2z7avfedkosP0Y2Ks01dYqQDichvuemkrgnDP+bk8uKqYE9Jj+emFM2ho62JOZlyf7ZOidTH5QKCJQKkx4LUt5WSNi2RWRuyg2+2tbcXR5SbEbuMnr+zAJlDd3MGp05IAmD3E/mps0s5ipUY5Ywx3vbiVB1ftOeKxVzaXcfdLW3G63ADsqvRcI/DwdfP5wbnTePyGhbjchodW7iE0yKYXjQUorREoNcqVNzpo7XRRUtPap7zD6eIXrxdQ29JBXEQIdy7LZVdFM3absGRyImfkpuB2G5KiQylvdDB/QhzBOqFcQNJXXalRrrDKM0X03kOtuN2fL+b3xtYKals6mJsZx0OrilldWMOuyiYmJUUSGmQHwGYTzpqeAsDsjLgjn1wFBE0ESo1ye6paAOh0uilraAc8zUV//c8+JidH8dzyxaTHhvHI6mIKKo6cRfScEzyJYN4ETQSBShOBUqNcUXVzz+2S2lb+s6eWZX9cw7ayRm5ckk1YsJ2rFk3gP3sOUdbQfsRUEadNTeKR6xZw/qw0X4euRghvrlmcKSIrRaRARHaIyHet8ntFpExENls/53srBqXGir21rezp9YHfW1F1C1OSowAoqWnhrpe20trp5HeXz+aqhZ4rgq9cmNkzRURuWt8OYRHh3BNStX8ggHnzlXcC3zfGTAcWA7eIyAzrsfuNMXOtnze9GINSY8IPX9jC9/+5ped+s6OLrzz0Ma9sLmNPVQuLc8YRHRrE+wVVHKxr5+ZTJ3FFXiY268M/JSaMM3OTAZiuC8yow3hzzeIKoMK63SwiBcB4bx1PqbHK5TbsKPcM+zTGICI8sHIPG/fXU1jZTHOHk6kpUeQkRfKfPYcAON360O/tzvNyycuKJyVGLxJTffmkLigiWXgWsl9nFd0qIltF5AkRifdFDEqNVvsPtdLW6aKt00Vlk4MDh9r469p9LJgYT3OHE4DJydFkJ0YCkJsazfi48COeZ1JSFMtPmYSIziKq+vJ6IhCRKOBF4HZjTBPwEDAJmIunxvCHAfZbLiL5IpJfU1Pj7TCVGrG6F4oBKKlp5dE1Jdhs8OC187nA6uCdkhJFTpKnn6C/2oBSg/FqIhCRYDxJ4BljzEsAxpgqY4zLGOMGHgUW9bevMWaFMSbPGJOXlJTkzTCVGtF2lvdOBC2s31vHouxxpMSE8atLZ/LEDXkkRoX2XBXcfV2AUsPlzVFDAjwOFBhj7utV3nuM2qXAdm/FoNRYsLOiidzUaCJD7Gw+2EhhdTMLJnhaVOMiQjgj1/PBf/b0FF69dQkLJmprqzo63pxiYglwHbBNRDZbZfcAV4vIXMAA+4BveTEGpUa9neVNnDwlkWC7jbe3V2AMzJ945MVfNpvo1cHqmHhz1NBaoL9eKR0uqtQw1TR3UN3cwYy0GFxuw7ayRkRgbqZ+4KvjRyedU2oE654tdEZaDK0dLgCmpUQTHRbsz7DUGKOJQKkRbP+hNgCykyKpa+sEYN4E7QNQx5deU67UCFZa306wXUiJDutZTP7E7AQ/R6XGGq0RKDWClda3MT4uHJtNmJwczau3LmFmuq4ipo4vTQRKjWAH69vJiI/oua+jgpQ3aNOQUiNYWX0bGfFHTheh1PGkiUCpEaq900VtS6cmAuV1mgiUGqHKGjwjhjITIobYUqkvRhOBUiPUwXrPspNaI1DepolAqRGqtM5TI+jdWayUNww7EYjIySLyDet2kohkey8spVRpfTshdhtJUbqQjPKuYSUCEfkZcCdwt1UUDPzdW0EpFWhcbkN7p6tPWWl9O+Pjw3uWm1TKW4ZbI7gUuAhoBTDGlAPRg+6hlBq2X76xkzP/sIoOpwuX2/D29kq2ljVo/4DyieFeUNZpjDEiYgBEJNKLMSkVUBrbu3hu/UHau1y8s6OKHWWNPLK6hCCbcP1JWf4OTwWA4SaCf4rII0CciHwTuBHP6mJKqQGsKaphTVEt95w/fdDtXtxYSnuXi9jwYB5eVUxxTQsXzUnnt5fPJizY7qNoVSAbVtOQMeb3wAt4lp2cBvzUGPN/3gxMqdHuzW2VPLqmhE6ne8Bt3G7D05/uZ/6EOL51ag47K5pwG8MPzp2mSUD5zJCJQETsIvK+MeY9Y8wPjDH/zxjzni+CU2o0a2rvwhioaGzHGENrh/OIbV7bWs7e2lZuWJLNFQsyCQ2ycfWiCXoRmfKpIROBMcYFtInIUU15KCKZIrJSRApEZIeIfNcqTxCR90SkyPqtk6urMamxvQvwjP55dUs5C3/1PmUN7T2PO7pc/Pbt3ZyQHsOFs9JIig7l/TtO5ccXzPBXyCpADXfUkAPP2sOPi8ifu3+G2McJfN8YMx1YDNwiIjOAu4APjDFTgA+s+0qNOZ8ngjY+O9BAW6eLpz/Z3/P4X/+zj7KGdn50wfSeIaKZCRGEBOl1nsq3httZ/Ib1M2zGmAqgwrrdLCIFwHjgYuA0a7OngFV4rlFQakzpXSMormkB4Nn1B/jumVMQgUfXlHDatCS+NCnRn2EqNbxEYIx5SkRCgKlW0W5jTNdwDyIiWcA8YB2QYiUJjDEVIpJ8VBErNUo0WEtLlta3U1zdQnZiJHtrW3lhUykhdqGutZPlp+T4OUqlhpkIROQ0PN/e9wECZIrI9caY1cPYNwrPaKPbjTFNIsO7SlJElgPLASZMmDCsfZQaKdxuQ7PVOby7spnyRgffP3sqqwpr+MVrO0mIDCE3NZqTcsb5OVKlht9H8AfgHGPMqcaYU4BzgfuH2klEgvEkgWeMMS9ZxVUikmY9ngZU97evMWaFMSbPGJOXlJQ0zDCVGhmaHU6M8dzeWdEEwOTkKB6/Po8TxsdQ2eTgppOzGe4XI6W8abiJINgYs7v7jjGmEM98QwMSzzv8caDAGHNfr4deBa63bl8PvDL8cJUaHbr7B8bHfT5FxOTkKOIiQvj7TSfy0LXzuWx+hr/CU6qP4XYW54vI48DT1v1rgY1D7LMEuA7PaKPNVtk9wK/xXKl8E3AAuOLoQlZq5OtOBDPSYyhraMduEyaM81wbEBkaxHmz0vwZnlJ9DDcRfBu4BbgNTx/BauDBwXYwxqy1tu3PmcMNUKmR7oa/rmdSUhQ/ufDz8f/dieCE9Bje21nFhIQIQoP0SmE1Mg03EQQBf+pu4hERO6CTpKuAZ4xh4756apo7+pR3J4KZ6Z7rMCclRfk8NqWGa7h9BB8AvefDDQfeP/7hKDW6NLZ30dzhpKSmFbfb9JQ3tHuGjk5LjSYs2EZuqs7arkau4dYIwowxLd13jDEtIqKToaiAd7DOM2VEe5eLiiZHT+dwd40gMSqUf9+ypE+nsVIjzXBrBK0iMr/7jojkAe2DbK9UQCitb+u5XVzd812JxvYuQuw2qzYQQ3TYoIPslPKr4dYIbgf+JSLlgAHSgSu9FpVSo8TB3omgpoXwEDutHU6a2ruICQ/W6wTUqDBoIhCRhcBBY8wGEckFvgVcBrwN7PVBfEqNaKX17USHBSHAnuoWnv5kP22dLuZPjCM2fLjfs5Tyr6Gahh4BOq3bJ+G5DuAvQD2wwotxKTUqHKxrIzM+gknJUby7s4qS2lYqmxzsrmwmNlybg9ToMFQisBtj6qzbVwIrjDEvGmN+Akz2bmhKjXyl9e1kJoQzKSmqzxDS4ppWTQRq1BgyEYhId/32TODDXo9pvVcFNGMMpfXtZMRH9FwnsHRKYs96AnERIf4MT6lhG+rD/FngIxGpxTNKaA2AiEwGGr0cm1Ij2qHWTtq7XGTGh5MR7xlN/ZX5GbR2ONl0oEFrBGrUGLRGYIz5FfB94EngZGO651PEBnzHu6EpNbJ8uKuKts7P1x0+WOcZMZSZEMFp05K4/8o5XDg7jXkTPKuvxmgiUKPEcNYs/tQY87IxprVXWaExZpN3Q1Nq5CioaOLGJ/P5V34pAO/vrOLBVcUAZMRHEGS3cem8DILsNuZmxgFojUCNGtrOr9QwrC6sAWB3VTOOLhc3/30jYcF2zshNJjsxss+2i7ITCAu2kXNYuVIjlSYCpfrx8EfFjIsM4Yq8TADWFNUCsKeqhaKqFpxuw28vn835/UwnnRITxsYfn01EiM42qkYHTQRKHcbR5eKP7xcSHxHC5QsycHS5Wb/PM4q6sLqZAmvFselpMQM+R2So/mup0WO4cw0pFTA27q/H0eWmotHB1tJG1u09RKfTzWnTkmho62LtnloiQuxMTNB5F9XYoIlAqcOsLqwh2C7YbcI7Oyr5oKCakCAb1yyaAMB7O6uYlhqNzabzCKmxwWv1VxF5ArgQqDbGzLTK7gW+CdRYm91jjHnTWzEodSxWF9WyYGI8dpvw7PoDNLR3cdm8DGZleBaZae9ykZs6cLOQUqONN2sETwLL+im/3xgz1/rRJKBGlOpmBwUVTZwyNYlzZqRS39bFzPRYfnnJTFJjwoiy2v5npOlCM2rs8FqNwBizWkSyvPX8SnnDJ8WHAFg6OYmJiRFUNjm4/qQswq0RQJOTo9h8sGHQjmKlRht/9BHcKiJbReQJEYn3w/GVGtC+Ws/VwlNTo4gJC+bOZbmkxob1PD4l2TOn0DRdelKNIb5OBA8Bk4C5QAXwh4E2FJHlIpIvIvk1NTUDbabUcVVa30ZKTCihQf1fA/D1k7L48QXTdcUxNab4dLCzMaaq+7aIPAq8Psi2K7DWPMjLyzMDbafU8VTW0N4zgVx/ZmXE9nQaKzVW+LRGICK9L8O8FNjuy+MrNZTS+nZdaF4FHG8OH30WOA1IFJFS4GfAaSIyF8+6x/vwLH2p1IjgchvKG9q5cPaR00YoNZZ5c9TQ1f0UP+6t4yn1RVU1OXC6zaBNQ0qNRXplsVKW0vp2ADLitWlIBRZNBEpZSus9Q0c1EahAo4lAKUt3jSBdO4tVgNFEoJSltL6N5OhQwoJ1HQEVWDQRqDHpxY2lnPmHVTi6XMPep7S+nfHaLKQCkCYCNSZ9drCe4ppW3i+oGnpjy1AXkyk1VmkiUGOTkSZTAAAWkklEQVSGy23ocrkBqGnuAOCFjaV9tulyufnbJ/soa2hnX20r333uMzYfbKCwqpkDdW1MS4nyddhK+Z2up6fGjF+9UcCW0gZe/PaXqLYSwerCGiobHT0Tx63cVc1PX9nB/7xZgF2E1k4Xmw7UMykpisiQIK49caI//wSl/EJrBGrM2Hignu1ljRhjqGnuYG5mHG4DD6wswu32TFe16UADwXbhrOkp5GUl8Ker5lJa386q3TX819Js4iND/PxXKOV7WiNQY4Ixhr01LXQ43Rxq7aS6uYPzZ6Uxa3wsT3+6n9rmTh64Zh6bDtQzIz2WB66Z37NvcU0rr24u46aTs/34FyjlP5oI1JhQ19pJk8MJwO7KZjqdbpKjQ7np5GxSY8P43Tu7eb+giq2lDVxtrT3c7Y6zp3L7mVN0DWIVsLRpSI0Je2tbe25vPtgAQFJ0KCLCt07JISk6lF+/tQtHl5t5E45cD0mTgApkmgjUmFBS83ki+OzA54kAIMhu47J549l3yDOFxPwJcb4PUKkRTBOBGpU27q9n1r3vcNZ9H/HP/IOU1LYSbBfCgm1sKfUkguToz5eYvCIvA/AkB11vQKm+tI9AjUqfHain2eEkKdrwi9d3Mn9CPBPHReI2pqd20F0jAJicHM2pU5NIjwtHRJuBlOpNawRqVCqtbycqNIhfXDyTZoeTjwpryE6M7Pm2HxpkIyas7/ecJ7+xkP+9bJY/wlVqRNNEoEalsgbPkpIn5YzrmTY6JymS9FjP7eSY0CO++WtNQKn+eS0RiMgTIlItItt7lSWIyHsiUmT9PnL4hgpY+2pb+fbfN9LY3jXktt0TxNlswhULMgHISYzsmUI6KSp0sN2VUr14s0bwJLDssLK7gA+MMVOAD6z7SgHwj/UHeGt7JS8eNj9QN6fLTXWTA4Cy+raemsDVJ2aydEoiSyYnkh7n6SDu3VGslBqc1xKBMWY1UHdY8cXAU9btp4BLvHV8NboYY3hzWwUAz204gDHmiG3++H4RZ/zhI6qbHTQ5nD39AcnRYTx904lkxEf0lPXuKFZKDc7XfQQpxpgKAOt3so+Pr0aoHeVNlNa3kzcxnsKqFjZZ1wJ0c3S5+Pu6/bR0OHlrWyVAv2sHpPUkB00ESg3XiO0sFpHlIpIvIvk1NTX+Dkd52VvbK7DbhPuvnEtEiJ1nPt3f5/FXNpfR0ObpO3htSzlAv9cDZMaHc+m88Zyeq98xlBouXyeCKhFJA7B+Vw+0oTFmhTEmzxiTl5SU5LMAle8drGvjxY1lLM5JIDMhgmsWTeDlzWVssaaKqGvt5Im1+8hNjSY7MZL8/fUA/S4iE2S3cf+Vc5k5Ptanf4NSo5mvE8GrwPXW7euBV3x8fOUDLR1O1pUcGvDxxrYu9llzAxVVNXPpg/+hrdPJ98+ZBsBtZ00hMSqUe17exj0vb2PJrz+ksLqZW06fzDxreojQIBuJUTpltFLHgzeHjz4LfAJME5FSEbkJ+DVwtogUAWdb99UY89e1e7nq0U+pskb49NbW6eSKRz7mogfW0t7p4qFVxXQ43bz030uYb00GFxMWzI/On86O8iZe2lTK+bPSeO97p/DlOek924zXK4SVOm68NsWEMebqAR4601vHVCPDltIGjPHMB3T+rLSecmMMP/73dgqrWgB4fWs57+6s4oJZaUxO7rtE5MVz05kwLoKpKdFEhX7+Nu1JBLrIvFLHzYjtLFaj17ayRsCTCHpbv7eOlzaVcdsZk0mLDeNXbxbQ0uHky3PSj3gOEWH+hPg+SQBgWmo0MWFBZCdGeu8PUCrA6KRz6qg8sXYvVU0OvjQ5kWkp0aQcNpVDdZODqibPesGHJ4Jn1x8gOiyIb582mS634aFVxSRGhXLSpHHDPr7dJvzr5i/pdQJKHUeaCNSwVTS288s3duI28MjqEsDTVn/1oky+deokgu22ntpA3sR4tpQ24OhyERZsp6Gtkze3V3LVwkzCQ+x8Zf54HlpVzPmzUrEf5aIw01Kjj/vfplQg00Sghu2fG0pxG3j79qXUtXSyp6aFd3ZU8vt3C4mNCOG6xRPZVtaICFx30kTyn6tnW1kjC7MSeGlTGZ1ON1ct9CwTOTk5mke/nseCiTrdlFL+pn0EakBri2r5/Tu7AXC5Dc9vOMDSKYnkpsbwpcmJfP2kLP5+04lMHBfByl2eS0K2lzUyKSmKpVM8136s31tHdZODhz4qZm5mHDPSY3qe/+wZKSRE6hBQpfxNE4Ea0JMf7+OBlXs4WNfGR4XVlDc6uOawhd9FhNOmJvFxcS2OLhfbyhqZNT6WhMgQ5k2I4/73Crlqxae0OJz85iuz/fSXKKUGo4lADWirteTjOzsq+ce6AyRGhXLWjJQjtjstNxlHl5t7X91BVVNHT+fvkzcs4vTcZEpqW/nfy2Zp275SI5T2Eah+VTY6qG72jP55dv0B9ta2crPVIXy4k3LGERpk47kNB5mRFsNl88YDEBsRzIrrFlDV1EFqrE4LrdRIpTUC1a/uBeBPnZpEcU0rbgNXH9Ys1C0s2N5TC/jFJScQ1CtZiIgmAaVGOE0Eql9bSxuw24TvnT0VgKVTEslMOHKSt24/PDeX+6+cw4KJCb4KUSl1nGjTkOrjsTUlHKhrY091C1NTopmTEcsdZ0/ljCGmdZ6RHtNnRJBSavTQRKAAzzxAnS43f/6giCaHE4CrFmYiItx25hQ/R6eU8iZNBIqrV3zKxHERnDU9hSaHk1OmJrG6sKZnymel1NimiSDAlda38UnJIT4pOcS2skZiw4N57Ot5bD7YwJxMXdxFqUCgncUB7oMCzxXB0WFB7ChvYtkJqYQE2ViUnUBokN3P0SmlfEETwRjW2uHk/vcKaet09pStLqzhj+8X4nYbAN4vqCInKZIfXzAdgIvnHTkltFJqbNOmoTHstS3l/OmDItLjwrhy4QTe3VHJLf/YRJfL0N7l4tbTJ/NpySFuXJLNV/MyWTAx4YgFYpRSY59fEoGI7AOaARfgNMbk+SOOsW51UQ0A7+2sZnHOOG75xyZmpMeSmxLNIx+V8PqWCrpchjOnpyAimgSUClD+rBGcboyp9ePxxzSny83aolpEYO2eGh7+KARj4JGvLSAxKoT4yBAO1rdxwew0nQpaqQCnTUOj1Mrd1dS1dPKVBRk9ZY+tKWFRdgKzM+LYUtpIk8PJ1YsyeXb9QZ5df5CL56b3TPdw13m5/gpdKTXC+Kuz2ADvishGEVnupxhGrbVFtXzzqXy+/68trNztGfVT09zBL98o4M4Xt+F2G1YX1mATuOPsaURb6/5+Y0m2P8NWSo1Q/koES4wx84HzgFtE5JTDNxCR5SKSLyL5NTU1vo9wBHG5DRWN7QAUVTVz8983Mjk5itzUaL7/zy1UNzlYt/cQAAUVTbywsZTXtpQzOyOOpOhQLs/L4MzcZOZm6gViSqkjiTHGvwGI3Au0GGN+P9A2eXl5Jj8/33dBjSDPrT/AAyv3UFrfzh1nT+Xfn5XR5HDy6q1LaOt0suyPa7hpaTatHU5e3lRGYnQo+w+1ERJk47Gv53HK1CR//wlKKT8RkY3DGYzj8xqBiESKSHT3beAcYLuv4xgN3t5eyV0vbSMpOpSzpqdw33uFHKhr48Fr55MeF87k5GhOm5bMS5vK+HjPIRZmJ3DXslySokN58oaFmgSUUsPij87iFOBlEek+/j+MMW/7IY4RraKxnbte2sqs8bE8v/wkgmzCXz/eR2pMGIuyP5/q+Yq8DN4vqKKmuYMr8jI5b1Yay2amYp1fpZQaks8TgTGmBJjj6+OONj97ZQcdXW7+dNVcQoI8FbebTj6ys/eM3GQSo0KobelkcY4nQWgSUEodDR0+OgI4ulz87ZN9VDV1kDUugonjInl3ZxU/OHcaOUmDX+QVbLdx1cIJPLfhIDPH6yRxSqmjp4lgBHhw5R7+/OEeIkLstHW6sAmMjwvvtwbQn++dPZWbT+t/PWGllBqKfnL42YFDbTy8uoSL56az4+fn8vDXFpCbGsMvL5lJWPDwZv+024SoUM3pSqljo58efmSM4d7XdhBkE+4+bzoiwrKZqSybmerv0JRSAURrBH700qYyPtxVzR1nT+2Z+kEppXxNE4GfVDS2c+9rO1iYFa9TPyil/EoTgY90udw9t50uN999djNOl+F3l8/BbtPhnkop/9E+Ai/rcrl5dE0Jf/6giNOnJXPbmVN4Zt1+1u+r4/4r55CVGOnvEJVSAU4TgRd1OF3811P5rCmqZVF2Ah8UVPPW9koAvrZ4ApfOyxjiGZRSyvs0EXwBuyubEfEM3/zsQAOOLhex4cFkxIfjdBseXV3CmqJa/veyWVy9aAJFVc18ureOE7MTmKKrgSmlRghNBAMwxnDDXzcwLTWau8/L7TNtw57qZn7+2k7WFA29wNqPL5jO1YsmADAlJZopKdFei1kppY6FJoIBbNxfz0eFNXxUWEN6bBiX52USHmynvKGdq1asw+l2c8/5uaTGhuPocjE3M47Y8GAa2ro4WNeG3S5MTYlmfFy4v/8UpZQalCaCAfwz/yARIXYWZiVw72s7ufe1nUSHBREaZKfT6eKl//4Sk5OP/HafEhPGtFT91q+UGj0CMhEYY9hb29ozoZvLbWjtdLK+pI7XtpYza3wsr2+t4MLZafzsyyfw8mdltHU62Vvbyo7yJn50/vR+k4BSSo1GAZkIHvhwD394r5A/XjmXSUlRXPfEOhraugCIDg3ilc3lAFy5MJPI0CC+tniiP8NVSimvGvOJoNnRxcfFhwgNsjFxXCQtDid//rAIu034+Ws7iIsIITzYzq0XTCYzIYIzcpP5uPgQhZXNzJ8Q7+/wlVLK68Z0IvjdO7tYsbqELlffdZkTo0J54Jp5XPf4Ohrau3j2m4tZnDOu5/FTpyZxqi7zqJQKEH5JBCKyDPgTYAceM8b82hvHmZoSzQ1fyuKs6SnYbML+Q20cONTKqdOSWTAxnvu+Opcul7tPElBKqUAjxpihtzqeBxSxA4XA2UApsAG42hizc6B98vLyTH5+vo8iVEqpsUFENhpj8obazh+Tzi0C9hhjSowxncBzwMV+iEMppRT+SQTjgYO97pdaZUoppfzAH4mgvzmXj2ifEpHlIpIvIvk1NTU+CEsppQKTPxJBKZDZ634GUH74RsaYFcaYPGNMXlKSjuBRSilv8Uci2ABMEZFsEQkBrgJe9UMcSiml8MPwUWOMU0RuBd7BM3z0CWPMDl/HoZRSysMv1xEYY94E3vTHsZVSSvWlaxYrpVSA8/kFZcdCRGqA/cewayIw9Oox/qGxHRuN7diN5Pg0tmMzVGwTjTFDjrYZFYngWIlI/nCuqvMHje3YaGzHbiTHp7Edm+MVmzYNKaVUgNNEoJRSAW6sJ4IV/g5gEBrbsdHYjt1Ijk9jOzbHJbYx3UeglFJqaGO9RqCUUmoIYzIRiMgyEdktIntE5C4/x5IpIitFpEBEdojId63ye0WkTEQ2Wz/n+zHGfSKyzYoj3ypLEJH3RKTI+u3zdTtFZFqv87NZRJpE5HZ/nTsReUJEqkVke6+yfs+TePzZeg9uFZH5fojtdyKyyzr+yyISZ5VniUh7r/P3sDdjGyS+AV9HEbnbOne7ReRcP8T2fK+49onIZqvcp+dukM+P4/u+M8aMqR8801YUAzlACLAFmOHHeNKA+dbtaDyL8swA7gX+n7/PlxXXPiDxsLLfAndZt+8CfjMCXtdKYKK/zh1wCjAf2D7UeQLOB97CM9vuYmCdH2I7Bwiybv+mV2xZvbfz47nr93W0/j+2AKFAtvX/bPdlbIc9/gfgp/44d4N8fhzX991YrBGMqIVvjDEVxphN1u1moIDRsf7CxcBT1u2ngEv8GAvAmUCxMeZYLiw8Lowxq4G6w4oHOk8XA38zHp8CcSKS5svYjDHvGmOc1t1P8cz06xcDnLuBXAw8Z4zpMMbsBfbg+b/2eWwiIsBXgWe9dfzBDPL5cVzfd2MxEYzYhW9EJAuYB6yzim61qm9P+KPppRcDvCsiG0VkuVWWYoypAM+bEUj2W3QeV9H3n3GknLuBztNIex/eiOebYrdsEflMRD4SkaX+Cor+X8eRdO6WAlXGmKJeZX45d4d9fhzX991YTATDWvjG10QkCngRuN0Y0wQ8BEwC5gIVeKqf/rLEGDMfOA+4RURO8WMsRxDPdOUXAf+yikbSuRvIiHkfisiPACfwjFVUAUwwxswD7gD+ISIxfghtoNdxxJw74Gr6fgHxy7nr5/NjwE37KRvy3I3FRDCshW98SUSC8byIzxhjXgIwxlQZY1zGGDfwKF6s+g7FGFNu/a4GXrZiqequUlq/q/0VH54EtckYUwUj69wx8HkaEe9DEbkeuBC41liNyFaTyyHr9kY8bfBTfR3bIK/jSDl3QcBlwPPdZf44d/19fnCc33djMRGMqIVvrDbGx4ECY8x9vcp7t9tdCmw/fF9fEJFIEYnuvo2ng3E7nnN2vbXZ9cAr/ojP0udb2Ug5d5aBztOrwNetURyLgcbuqryviMgy4E7gImNMW6/yJBGxW7dzgClAiS9js4490Ov4KnCViISKSLYV33pfxwecBewyxpR2F/j63A30+cHxft/5qvfblz94es4L8WTrH/k5lpPxVM22Aputn/OBp4FtVvmrQJqf4svBM0JjC7Cj+3wB44APgCLrd4Kf4osADgGxvcr8cu7wJKMKoAvPN6+bBjpPeKrof7Heg9uAPD/EtgdPe3H3++5ha9uvWK/1FmAT8GU/nbsBX0fgR9a52w2c5+vYrPIngZsP29an526Qz4/j+r7TK4uVUirAjcWmIaWUUkdBE4FSSgU4TQRKKRXgNBEopVSA00SglFIBThOBGtNExCV9ZzAddDZaEblZRL5+HI67T0QSj2G/c61ZOeNF5M0vGodSwxHk7wCU8rJ2Y8zc4W5sjPH6lMxDWAqsxDMj5n/8HIsKEJoIVEASkX14pg443Sq6xhizR0TuBVqMMb8XkduAm/HM07PTGHOViCQAT+C5EK8NWG6M2Soi4/BcmJSE5ypY6XWsrwG34ZkWfR3w38YY12HxXAncbT3vxUAK0CQiJxpjLvLGOVCqmzYNqbEu/LCmoSt7PdZkjFkEPAD8sZ997wLmGWNm40kIAD8HPrPK7gH+ZpX/DFhrPJORvQpMABCR6cCVeCb2mwu4gGsPP5Ax5nk+nxN/Fp7pFuZpElC+oDUCNdYN1jT0bK/f9/fz+FbgGRH5N/Bvq+xkPNMMYIz5UETGiUgsnqacy6zyN0Sk3tr+TGABsMEzbQzhDDyB3xQ8UwMARBjP/PNKeZ0mAhXIzAC3u12A5wP+IuAnInICg0/z299zCPCUMebuwQIRzxKhiUCQiOwE0sSzPOJ3jDFrBv8zlPpitGlIBbIre/3+pPcDImIDMo0xK4EfAnFAFLAaq2lHRE4Dao1nfvje5ecB3YusfABcLiLJ1mMJIjLx8ECMMXnAG3j6B36LZ/K/uZoElC9ojUCNdeHWN+tubxtjuoeQhorIOjxfiK4+bD878Her2UeA+40xDVZn8l9FZCuezuLuqYB/DjwrIpuAj4ADAMaYnSLyYzwrwNnwzHB5C9Dfkpvz8XQq/zdwXz+PK+UVOvuoCkjWqKE8Y0ytv2NRyt+0aUgppQKc1giUUirAaY1AKaUCnCYCpZQKcJoIlFIqwGkiUEqpAKeJQCmlApwmAqWUCnD/P48nAybeD/rDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x144ed1d0>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(1, len(scores)+1), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
